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EXECUTIVE SUMMARY 



The Job Corps program has been a central part of federal efforts to provide employment 
assistance to disadvantaged youths since 1964. Job Corps serves economically disadvantaged youths 
between the ages of 16 and 24 who can benefit from a wide range of education, vocational training, 
and support services in a predominantly residential setting. Currently, 116 Job Corps centers operate 
nationwide, serving more than 60,000 new enrollees each year, at an annual cost of more than 1 
billion dollars. Given the program’s size and its central role in federal efforts to assist disadvantaged 
youths, a comprehensive evaluation of the program is an important priority. 

The National Job Corps Study, funded by the U.S. Department of Labor (DOL), is designed to 
provide a thorough and rigorous assessment of the impacts of Job Corps on key participant 
outcomes. An analysis of program benefits and costs and a process study are also being conducted. 
The cornerstone of the study is the random assignment of all program applicants found eligible for 
Job Corps to either a program group or a control group. Program group members were permitted 
to enroll in Job Corps. Control group members were not permitted to enroll in Job Corps for a 
period of three years (although they could enroll in other training or education programs). Program 
impacts will be estimated using follow-up survey data collected 12, 30, and 48 months after random 
assignment, as well as administrative records data. 

This report describes the implementation of random assignment and sample intake, presents 
evidence that the process was implemented in a way that will enable the study to realize its goals, 
and draws lessons from the experience that may be applicable to other program evaluations. 



STUDY DESIGN 

The Job Corps evaluation is based on a national sample of eligible program applicants. Youths 
were sampled from all outreach and admissions (OA) agencies nationwide between November 1994 
and February 1996. This nonclustered design was adopted because the national sample will produce 
more precise impact estimates than a clustered design of the same size, and because this approach 
spreads the burden of random assignment across all OA agencies and Job Corps centers. 

Youths were randomly assigned after they were determined to be eligible for the program and 
were ready to be, but had not yet been, assigned to a center. This point in the Job Corps intake 
process was chosen for two reasons. First, it addresses a useful and well-defined policy question: 
What are the effects of Job Corps on youths who apply for and are found eligible for Job Corps? 
Second, random selection procedures could be incorporated into the existing intake process, with 
acceptable levels of disruption. 

Overall, the sampling rate to the control group was 7.4 percent on average. It was set lower for 
females who had a high likelihood of being a residential student because residential females are 
difficult to recruit and Job Corps staff were concerned that the study would cause slots for residential 
females to go unfilled. 
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IMPLEMENTATION OF SAMPLING PROCEDURES 



A well-implemented random assignment study requires consistently accomplishing three tasks: 



1 . Explaining the study to prospective program applicants 

2. Ensuring that all people in the population of interest are subject to random selection and 
that each person is subject to it once and only once 

3. Ensuring that only people randomly selected to the program group enroll in the program 



To ensure that these tasks would be accomplished with minimum burden on OA staff, the study 
team investigated OA procedures in each region and developed proposed procedures for conducting 
random assignment tailored to each region. With assistance from Job Corps regional office staff, 
we then met with senior representatives of each organization that conducted outreach and admissions 
in each region. These meetings were used to discuss why random assignment was necessary and 
then to refine the proposed procedures for conducting random assignment to be sure they worked 
for staff in the region. These meetings helped OA managers to think concretely about how the need 
to form a control group who could not enroll in Job Corps would affect their staff. The meetings 
also produced several specific suggestions for materials that would assist OA staff in presenting the 
study. 

In late summer and fall 1 994, the study team conducted training sessions for nearly all the OA 
counselors and coordinators in each Job Corps region. Approximately 900 OA staff from 100 OA 
agencies attended the sessions, which were designed to inform Job Corps staff about the reasons for 
the study and to provide them with the information necessary to perform their study-related tasks. 

After a brief period for testing procedures beginning on November 1, 1994, sample selection 
began on November 17, 1994 and continued through February 28, 1996. During this period, OA 
staff were required to submit information to MPR for all new eligible applicants before the applicant 
could be assigned to a Job Corps center. All eligible Job Corps applicants whose application date 
for Job Corps was between November 17, 1994, and December 16, 1995 were subject to selection 
for the study control group. For applications that MPR received from December 17, 1995, to 
February 28, 1996, only people whose application date was before December 17 were part of the 
sample and subject to random selection. 

The core random assignment process consisted of four steps: 



1 . Job Corps OA staff informed each Job Corps applicant about the study. 

2. For each new applicant in the sample frame, Job Corps OA staff completed and 
transmitted three forms to MPR: the Job Corps application form, a study-specific 
supplement to the application form, and an Agreement to Participate form. 
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3. MPR checked that all key information for random assignment was complete, that 
applicants were in the sample frame, and that they had not previously been sent for 
random assignment. Then, each new applicant in the sample frame was randomly 
assigned to the control, program research, or program nonresearch group. 

4. MPR notified Job Corps staff of the random assignment results within 48 hours, and sent 
an official notification letter signed by DOL officials to control group members. Most OA 
staff also contacted youths they recruited about the random assignment results. 



Job Corps staff assigned only program group members to a center slot. By checking a study 
form completed for each applicant, center staff determined that each incoming student had been sent 
to MPR for the random selection process and had not been assigned to the control group. 

Over 1,300 Job Corps OA counselors nationwide were directly involved in random assignment 
during the sample intake period, and approximately 1 1 0 Job Corps OA coordinators and approvers 
transmitted materials to MPR. During the sample intake period, nearly 81,000 applications in the 
sample frame were processed by MPR. The final sample consists of 5,977 control group members, 
9,409 program research group members, and 65,497 program nonresearch group members. 



MONITORING SAMPLE BUILDUP AND ENDING RANDOM ASSIGNMENT 

During the sample intake period, MPR staff monitored sample buildup to assess whether the 
research sample was near target levels and whether initial sample design parameters needed to be 
adjusted. This monitoring process also guided plans for ending random assignment because we wanted 
to end sample intake only after the research sample size targets were attained. 

By mid- 1995, the cumulative number of eligible Job Corps applicants sent for random assignment 
was well below the levels anticipated on the basis of historical data, and centers were operating well 
below full capacity. Three factors appear to be responsible for the shortfall in applicants. First, in 
March 1995, Job Corps instituted several major changes in program policies (for example, 
strengthening zero tolerance policies for violence and drugs) that temporarily disrupted flows into the 
program. Second, the Job Corps program received significant negative publicity during late 1 994 and 
early 1995. Finally, the presence of the control group for the National Job Corps Study contributed 
to the shortfall, as the outreach system was initially not able to increase the numbers of eligible 
applicants as planned. 

Because of the shortfall, initial plans to end random assignment in late fall 1995 were revised, 
and sample intake was extended until early 1996. Beginning in summer 1995, the outreach and 
intake system began a concerted effort to bring centers to full capacity. This led to a surge in new 
applications in late summer and fall 1995, which allowed sample size targets to be met and exceeded 
by the end of the year. Sample intake ended on February 29, 1996, although only those eligible 
applicants who applied to Job Corps before December 17, 1995, were included in the sample frame. 
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MONITORING ADHERENCE TO RANDOM ASSIGNMENT PROCEDURES 



Job Corps staff implemented the random assignment procedures successfully over the 16-month 
sample intake period. Less than 0.6 percent of youths in the sample frame were not randomly 
assigned, and we estimate that very few youths who are outside of the sample frame are in the 
sample. In addition, through the end of February 1 999, just 1 .4 percent of control group members 
enrolled in Job Corps before the end of the three-year period during which control group members 
were not supposed to enroll. Hence, we believe that the research sample is representative of the 
youths in the intended sample frame and that the bias in the impact estimates due to contamination 
of the control group will be small. 

The Job Corps Student Pay, Allotment, and Management Information System (SPAMIS) has 
enabled MPR to identify center enrollees in the sample frame who were not randomly assigned and 
those who were previously assigned to the control group. MPR receives information on all new 
enrollees in Job Corps each week and matches this information with that for youths who were sent 
for random assignment. Early discovery of errors allowed the study team and Job Corps to take 
prompt corrective action. OA staff had lists of control group members showing the date on which 
each can enter, and SPAMIS incorporates a check when a center adds a student to the data system. 



EFFECTS OF THE STUDY ON PROGRAM OPERATIONS 

Overall, the study had noticeable effects on key aspects of program operations. Job Corps 
experienced a decline in program intakes during the first half of the study intake period and a very 
large decline in on-board strength (OBS)-ffom 96 percent in January 1995 to under 80 percent in 
July 1995. The study appears to have played a relatively modest role, with removal of control group 
members from the flow of applicants into the program accounting for approximately one-fourth of 
the drop in OBS. 

The effects of the study on OA counselors’ activities and the composition of students coming 
to the program appear to have been modest. Few said they started new outreach activities, spent 
more time on outreach, or lost referral sources because of the study. Most said the study had no or 
only small effects on their ability to recruit students, although one-third of students were recruited 
by OA counselors who said the study caused them significant problems that made recruiting more 
difficult. OA counselors reported that few students were dissuaded from applying or decided to 
postpone their application because of the study’s random selection procedures. Finally, OA 
counselors do not appear to have provided substantially more assistance in finding alternative 
training opportunities to the control group than they provided for other applicants who could not 
enroll in Job Corps. 



LESSONS 

We believe that the implementation of the National Job Corps Study offers three lessons for the 
successful implementation of a randomized study design in an ongoing program: 
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1 . The active, visible commitment of program managers to the success of the study is 
very important Job Corps managers wanted a well-implemented study because they 
believed a strong study would demonstrate that their program is effective, and thereby 
engender continued public support for it. Program managers effectively communicated 
this message to program staff. Believing their study-related tasks were important, 
program line staff performed diligently the tasks of telling applicants about the study, 
gathering necessary information, and making sure that only program group members are 
sent to Job Corps. 

2. Research staff should work closely and continuously with the line staff who conduct 
program outreach and intake. This entails making sure line staff understand why 
random assignment is necessary; making study-related tasks of line staff as simple as 
possible; providing staff with appropriate materials to help them explain the study to 
applicants and the public; training staff to perform their study-related tasks, and 
providing ongoing technical assistance to program staff. 

3. Monitoring entry into the program ensures the integrity of the study. Maintaining 
study integrity is essential for ensuring that staff’s efforts are not wasted. It allows 
problems to be identified and corrected quickly. 
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I. INTRODUCTION 



Job Corps plays a central role in federal efforts to provide employment assistance to 
disadvantaged youths. The program’s goal is to help disadvantaged youths become “more 
responsible, employable, and productive citizens” by providing comprehensive services, including 
basic education, vocational skills training, counseling, and residential support. It serves more than 
60,000 new enrollees each year at an annual cost of more than $1 billion. The National Job Corps 
Study, funded by the U.S. Department of Labor (DOL), was designed to provide information about 
the effectiveness of Job Corps in attaining its goal. 1 The cornerstone of the study is the random 
assignment of all youth found eligible for Job Corps either to a program group, in which they were 
permitted to enroll in Job Corps, or to a control group, in which they were not. 

Implementing random assignment nationally in an ongoing program presents challenges. Care 
is necessary to ensure that the way random assignment is implemented neither compromises the 
ability of the study to provide valid estimates of the impact of Job Corps nor places undue burden 
on program staff or applicants. Our monitoring of the process suggests that Job Corps staff 
implemented random assignment procedures very well. Only about 0.6 percent of the intended study 
population were not randomly assigned, and, so far, only 1.3 percent of control group members have 
enrolled in Job Corps. This report describes the way random assignment was implemented in this 
study, the evidence that it was implemented successfully, and the lessons we learned that may be 
applicable to other program evaluations. 2 



'The study is being conducted by Mathematica Policy Research, Inc. (MPR) and its 
subcontractors, Battelle Human Affairs Research Centers and Decision Information Resources. 

2 Burghardt et al. (1994) discuss in detail the design of the National Job Corps Study. 



A. OVERVIEW OF JOB CORPS 



The Job Corps program was established by the Economic Opportunity Act of 1964. In 1969, 
control of the program was transferred from the Office of Economic Opportunity to DOL. Job Corps 
was eventually incorporated without changes as Title IV in the Comprehensive Employment and 
Training Act (CETA) as enacted in 1973 and as amended in 1978, and then into the Job Training 
Partnership Act (JTPA) of 1982. 

Even though Job Corps is one of the most centralized of the DOL programs administered under 
JTPA, its operational structure is complex. Job Corps encompasses multiple levels of administrative 
responsibility, several distinct program components, and numerous contractors and subcontractors. 
Although many other employment and training programs have been decentralized, Job Corps is still 
administered primarily at the federal level. DOL administers Job Corps through a national office and 
nine regional offices. The national office establishes policy and requirements, develops curricula, 
and oversees major program initiatives. The regional offices procure and administer contracts and 
perform oversight activities, such as reviews of center performance. 

DOL contracts out center operations, recruiting and screening of new students, and placement 
of students in jobs and other educational opportunities after they leave the program. At the time of 
the study, Job Corps operated 1 10 centers nationwide. The U.S. Departments of Agriculture and 
Interior operated 30 centers, called Civilian Conservation Centers (CCCs), under interagency 
agreements with DOL. The other 80 centers were operated by private contractors selected through 
a competitive bidding process and are administered through contracts with Job Corps’ regional 
offices. Recruitment and placement are also administered through competitively awarded contracts 
with the regional offices. 
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1. Outreach and Admissions 



Recruitment and screening for Job Corps are conducted by Outreach and Admissions (OA) 
agencies, which include private nonprofit firms, private for-profit firms, state employment agencies, 
and the centers themselves. These agencies provide information to the public through outreach 
activities (for example, by placing advertisements and making presentations at schools), screen 
youths to ensure that they meet the eligibility criteria, sometimes assign youth to centers, and arrange 
for transportation to centers. 

To participate in Job Corps, youth must be legal U.S. residents ages 16 to 24. Males 18 or older 
must be registered with the Selective Service Board, and minors must have the consent of a parent 
or guardian. Youth must also be disadvantaged (defined as living in a household that receives 
welfare or has income below the poverty level) and living in a debilitating environment that 
substantially impairs prospects for participating in other programs. Youth must need additional 
education, training, and job skills and possess the capacity and aspirations to benefit from Job Corps. 
They must also be free of serious behavior and medical problems, and they must have arranged for 
adequate child care when they participate in Job Corps. 

2. Job Corps Services 

Job Corps is a comprehensive and intensive program. Major Job Corps components include 
basic education, vocational training, health care and education, residential living (including social 
skills training), counseling, and job placement assistance. Services in each of these components are 
tailored to each participant. 

Education. The goal of the education component is to enable students to achieve educational 
attainment as fast as their individual abilities permit. Education programs in Job Corps are 
individualized and self-paced and operate on an open-entry and open-exit basis. The programs 



include remedial education (emphasizing reading and mathematics), world of work (including 
consumer education, driver education, home and family living, health education, and programs 
designed for individuals whose primary language is not English), and a General Education 
Development (GED) program of high school equivalency for students who are academically 
qualified. Some centers also offer some students the opportunity to attend postsecondary education 
while enrolled in Job Corps. Students are assigned to classes based on the results of diagnostic tests 
administered during the first few weeks. 

Vocational Training. As with the education component, the vocational training programs at 
Job Corps are individualized and self-paced and operate on an open-entry and open-exit basis. Each 
Job Corps center offers training in several vocational trades, typically including business and clerical 
occupations, health occupations, construction trades, culinary arts, and building and apartment 
maintenance. National labor and business organizations provide vocational training at many centers. 
In many trades, students gain hands-on experience by working on supervised work projects, such as 
the construction or rehabilitation of buildings either on center or in the community. 

Health Care and Education. Students receive comprehensive health services, including 
medical examinations and treatment; immunizations; dental examinations and treatment (for 
participants who remain in the program at least 90 days); counseling for emotional and other mental 
health problems; and instruction in basic hygiene, preventive medicine, and self-care. 

Residential Living. Residential living is the most distinctive component of the Job Corps 
program and distinguishes it from most other employment and training programs. The idea behind 
residential living is that, given the disadvantaged environments from which most participants come, 
the students require a new and more supportive environment to derive the maximum benefits from 
education and vocational training. All students must participate in formal social skills training 



activities. The residential living component also includes meals, dormitory life, entertainment, 
sports and recreation, center government, center maintenance, and other related activities. 
Historically, regulations had limited the number of slots that can be reserved for nonresidential 
students to 10 percent, although the JTPA amendments that became effective in July 1993 raised that 
limit to 20 percent. 

Counseling and Other Ancillary Services. Job Corps centers provide counselors and 
residential advisers. These staff help students plan their educational and vocational curricula, offer 
motivation, and create a supportive environment. Support services are also provided during 
recruitment, placement, and the transition to regular life and jobs after Job Corps. 

Placement. The final step in the Job Corps process is placement. The placement component 
focuses on helping students find jobs in training-related occupations with prospects for long-term 
employment and advancement. Placement contractors are state employment offices or private 
contractors, and some centers perform placement activities. Placement agencies help students find 
jobs by providing interviewing and resume-writing assistance and job development and referral 
services. They are also responsible for distributing the readjustment allowance, a stipend students 
receive after leaving Job Corps. 

3. Recent Job Corps Policy Changes 

In response to congressional concerns about the operation of the Job Corps program, new 
policies were instituted between March and July 1995~during the sample intake for the study. These 
included introducing a “zero tolerance” (ZT) policy for drugs and violence and a “one strike and 
you’re out” rule to govern terminations of students found guilty of offenses prohibited under the zero 
tolerance policy. Terminations of students who test positive for drugs or demonstrate behavior 
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inconsistent with Job Corps zero tolerance for violence policy within 30 days of enrollment do not 
affect a center’s performance record. 

B. OVERVIEW OF THE NATIONAL JOB CORPS STUDY 

The study is addressing the following research questions: 

• How effective is Job Corps overall at improving the employability of disadvantaged 
youth? 

• Is Job Corps more or less effective for certain groups of the eligible population? 

• What is the Job Corps program “model,” and how well is the model implemented in 
practice? 

• What components of Job Corps (such as residential and nonresidential services and 
contract centers and CCCs) are particularly effective? 

• Is Job Corps cost-effective? 

To address these questions, the study consists of an impact analysis, a process analysis, and a 
benefit-cost analysis. We describe each component next. 

1. Impact Analysis 

The purpose of the impact analysis is to estimate the net impact of Job Corps on participants’ 
postprogram earnings and other employment-related outcomes. 

DOL structured the project so that careful consideration would be given in the design phase to 
whether the study should use random assignment to measure program impacts. Congress had 
directed DOL, through provisions of the JTPA, to evaluate its training programs using random 
assignment methods where feasible. The findings of studies using methods other than random 
assignment to create a comparison group, including a previous study of Job Corps, have been subject 
to question because of uncertainty about whether the experience of the comparison group provides 



a valid indication of what the experience of program participants would have been had they not had 
the opportunity to enroll in Job Corps. Random assignment avoids this problem. Moreover, 
beginning in the mid-1980s, DOL contracted for a large study of programs funded under Title II-A 
of JTPA that used random assignment methods and demonstrated the feasibility of using these 
methods to study ongoing programs. 

Yet Job Corps staff were justifiably concerned about the potential burden on individual Job 
Corps applicants who were selected for a control group and about the public relations problems that 
would follow from a study design in which some eligible applicants were not permitted to enroll in 
Job Corps. However, Job Corps serves only a small fraction of the eligible population. A large pool 
of unserved applicants could potentially be tapped to create a control group without reducing the 
number of youths served by Job Corps. In light of these circumstances and the need for reliable, 
credible information about program impacts, a study advisory panel, which included representatives 
of Job Corps, concluded that a random assignment design was feasible and should be used for the 
study. 

Between November 1994 and February 1996, approximately 6,000 Job Corps-eligible 
applicants residing in the contiguous 48 states and the District of Columbia were selected randomly 
for a control group. Control group members were not permitted to enroll in Job Corps for a period 
of three years, although they were able to enroll in other programs available to them. To maximize 
the efficiency of the sample, keep the burden on individual recruiting agencies low, and minimize 
threats to the validity of the evaluation, the control sample was selected from among all new, eligible 
applicants nationwide. 3 In this way, the burden of the evaluation was spread across all OA agencies. 



3 There are some exceptions; these are discussed in Chapter II. 



Approximately 1 eligible applicant in 14 (seven percent of 81,000 eligible applicants) was assigned 
to the control group. 

During the same 1 6-month period, about 9,500 eligible applicants assigned to Job Corps were 
selected for the research sample as members of the program group. This sample includes youth who 
enroll in Job Corps (about 70 percent of eligible applicants), as well as those who do not enroll, the 
so-called “no-shows” (about 30 percent of eligible applicants). Although the study’s research 
interest focuses on enrollees, all youth who were randomly assigned, including those who do not 
enroll at a center, will be included in the analysis to preserve the benefits of the random assignment 
design. 

We will examine five types of outcome measures: (1) employment and earnings; (2) education 
and training; (3) dependence on welfare and other public transfers; (4) antisocial behavior, such as 
arrests, crimes committed by and against sample members, and alcohol and drug use; and (5) family 
formation and childbearing. Impacts will be estimated for subgroups of youths, including those 
defined by the following baseline characteristics: age, gender, educational attainment, parental 
status, employment experiences, participation in welfare programs, and previous involvement with 
the law. 

To estimate the impacts of the Job Corps residential component, we will compare the 
experiences of program and control group youth who, before random assignment, were expected to 
be assigned to a residential slot. The impacts of the nonresidential program will be estimated by 
comparing the experiences of program and control group youth who were expected to be assigned 
to a nonresidential slot. Impact estimates will be obtained in a similar way for those designated for 
(1) CCC or contract centers; (2) low-, medium-, or high-performing centers; and (3) small, medium- 
sized, or large centers. Measurements of the impacts of other components (for example, specific 
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occupational training courses and duration of stay in Job Corps) will rely on statistical models of the 
process by which students are assigned to these components. 

We plan to collect survey data on members of the research sample at four points: 

1 . At baseline, immediately after random assignment 

2. 12 months after random assignment 

3. 30 months after random assignment 

4. 48 months after random assignment 

At baseline, we attempted telephone interviews with all sample members and in-person 
interviews with a random clustered subsample of those sample members who did not complete a 
telephone interview. The target sample for the 12-month follow-up interview includes (1) all sample 
members eligible for in-person interviews at baseline, and (2) those not eligible for in-person 
interviews at baseline who completed the baseline interview by telephone. In the 30-month and 48- 
month follow-up interviews, we will attempt to conduct interviews with all sample members who 
have completed at least one interview. In the 12-month follow-up interview, we first attempted to 
interview each member of the target sample by telephone. When these attempts were unsuccessful, 
we attempted to conduct the interview in person. A similar interviewing strategy will be used for 
the remaining follow-up interviews. 

We also plan to collect additional data on sample members. We plan to collect administrative 
data on social security earnings on all sample members, and earnings data from Unemployment 
Insurance (UI) administrative records on sample members in 17 randomly selected states. These data 
will be used to assess (1) whether nonresponse to the baseline and follow-up surveys affects survey- 
based earnings impact estimates, and (2) whether administrative and survey earnings data yield 



similar estimates of program impacts on earnings. In addition, we will administer basic skills tests 
to a subsample of the research sample in conjunction with the 30-month follow-up interview. 

2. Process Analysis 

The process analysis documents the Job Corps program model, assesses how the Job Corps 
program model is actually implemented in practice, and identifies important variations in program 
elements across centers or agencies that might affect student outcomes. 4 It also provides data that 
can be used to improve Job Corps operations and help develop other training programs for 
disadvantaged youths. Data collected for the process analysis will play a significant role in the 
analyses of the component and subgroup impacts and will enhance the interpretation of findings from 
both the impact and benefit-cost analyses. 

Data for the process analysis are from three sources: 



1 . Site Visits and Interviews. Weeklong site visits to a representative sample of 23 Job 
Corps centers, telephone interviews with staff at a linked sample of representative OA 
and placement agencies, and interviews with national office and regional office staff. 

2. Mail Surveys of Job Corps Centers. This nationwide mail survey will enable us to 
develop comprehensive measures of key program characteristics for all Job Corps 
centers. 

3. Program Administrative Records. These include data on participants’ experiences 
maintained in the Job Corps Student Pay, Allotment, and Management Information 
System (SPAMIS) and in reports submitted by contractors to the national and regional 
offices. 



3. Benefit-Cost Analysis 

The primary purpose of the benefit-cost analysis is to assess whether the benefits of Job Corps 
are commensurate with the substantial public resources invested in the program. The benefit-cost 




4 See Johnson, Terry, et al. “National Job Corps Study: Report on the Process Analysis.” 
Princeton, NJ: Mathematica Policy Research, Inc., February 1999. 
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analysis provides a unified, consistent framework for weighing the many potential benefits and costs 
of the program, including those that cannot be measured in dollars. By examining costs from the 
perspective of participants, nonparticipants, and the government, as well as from that of society as 
a whole, the benefit-cost framework provides information about the distribution of benefits and 
costs. 

The most important benefits that will be valued are as follows: 

• Increased output that may result from the additional employment and productivity of 
youth who have participated in Job Corps 

• Increased output produced by the youth while in Job Corps 

• Reduced criminal activity 

• Reduced use of other services and programs, including welfare and other education and 
training programs 

Other benefits to society that are difficult to appraise accurately include improvements in 
participants’ quality of life, self-esteem, health, and relationship skills, and reduction in crimes 
committed against participants. These benefits will be considered qualitatively. 

The most important costs of Job Corps include the following: 

• Program operating costs 

• Opportunity cost of attending Job Corps (primarily the earnings foregone while the 
student attends Job Corps) 

C. ORGANIZATION OF THE REPORT 

The successful implementation of random assignment is critical to obtaining valid estimates of 
the impact of Job Corps on the youth that it serves. In the rest of the report, we describe how random 
assignment was successfully implemented in the National Job Corps Study and how the study 



affected program operations. We begin in Chapter II by describing key design issues that arose in 
implementing random assignment, including the point in the intake procedures that random 
assignment took place, the sample frame, the planned sampling rates, informed consent procedures, 
and the need for key data to be collected prior to random assignment. Chapter III describes how 
random assignment procedures were implemented, including the process of planning for the 
implementation, the roles and responsibilities of Job Corps staff and MPR, and the way procedures 
were adjusted to changes during the intake period. Chapter IV describes how we monitored the 
sample buildup to ensure that we would obtain a sufficient sample. It also describes the results of 
the monitoring and our need to change the sampling rates and lengthen the intake period. Chapter 
V describes how we monitored the integrity of random assignment and provides evidence that 
program staff implemented random assignment well. Chapter VI discusses the impacts of the study’s 
random assignment procedures on Job Corps program operations. Lessons about the implementation 
of random assignment learned from this study and possible implications for other studies are 
discussed in Chapter VII. 
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II. DESIGN ISSUES 



The central objective of the National Job Corps Study is to estimate the impact of Job Corps on 
the earnings and other employment-related outcomes of program participants. Because a randomized 
design is most likely to produce valid impact estimates, the evaluation is based on random 
assignment. 1 Our challenge was to implement random assignment in a way that maintained the 
ability of the evaluation to provide valid estimates of the impacts of Job Corps while minimizing the 
impact of the study on the program staff and applicants. This chapter discusses some issues that 
relate to the implementation of random assignment and that affected the study design. 

The chapter begins by describing the point at which random assignment occurred in the Job 
Corps application process. Section B describes the choice of the sample frame. The planned 
sampling rates into the control and program research groups are described in Section C. Section D 
discusses obtaining informed consent from study participants. Finally, Section E discusses data we 
collected from applicants prior to random assignment. 

A. POINT OF RANDOM ASSIGNMENT 

The point in the Job Corps intake process at which random assignment is performed has 
important implications both for the quality and type of information that can be derived from the 
study and for its operational viability. We performed random assignment on all Job Corps applicants 
after they were determined to be eligible for the program and were ready to be, but had not yet been, 



'Job Corps staff frequently asked why we did not use eligible youth who did not enroll in the 
program, the so-called “no-shows,” as a comparison group. We rejected this design because no- 
shows are likely to differ from Job Corps enrollees in unobserved ways that are correlated with 
outcomes of interest. 
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assigned to a slot at a center. This section describes the rationale for choosing this point of random 
assignment. We begin by providing an overview of the intake process. 



1. Overview of the Job Corps Intake Process 

OA agencies recruit and screen applicants for Job Corps. OA counselors have the most direct 
contact with youth who apply to Job Corps. When random assignment was implemented, OA 
counselors used the Job Corps ETA-652 application form to collect information from the applicant 
(over the telephone or in person). 2 OA counselors are also responsible for obtaining any additional 
information from appropriate authorities needed to determine an applicant’s eligibility (for example, 
when an applicant has a history of health or behavioral problems). All this information is used to 
determine Job Corps eligibility. 

The organization or individual responsible for determining eligibility of youth who apply to Job 
Corps varies by region and, within a region, even by center. Depending on the situation, OA, center, 
or regional office staff may have responsibility for eligibility determination. First, applicants 
identified as having health or behavioral problems require a special medical, behavioral, or mental 
health review before they can be determined eligible for Job Corps. Typically, either regional office 
or center staff review these cases and make eligibility decisions. Second, in some centers in certain 
regions, OA staff located at the center where the student will be assigned are responsible for final 
determination of eligibility, even if they did not recruit the applicant. This latter type of case applies 
to “regular” applicants as well as to those requiring a special medical, behavioral, or mental health 
review. 

After applicants are determined eligible, OA counselors typically notify them of their acceptance 
to Job Corps and complete any remaining enrollment requirements. The youth are usually then 



Currently, many OA counselors input information using an electronic data-entry program. 
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assigned to a center, given a departure date, and provided with arrangements (for example, a bus 
ticket) for travel to the center. The time between when an applicant is determined eligible and when 
the applicant is assigned to a center for enrollment varies over time, by region, and by type of 
student. At times, some students (especially male students seeking residential slots) in some regions 
have waited for many months before assignment to a center. 

2. Issues Considered in Choosing the Point of Random Assignment 

We considered four possible points of randomization: 

1. At Application. Random assignment would occur after an application form was 
completed. 

2. After Applicant Eligibility Is Determined— Our Approach. Random assignment would 
occur as applicants entered a queue for assignment to a center slot. 

3. After an Open Slot Is Assigned. Random assignment would occur after applicants 
leave the queue for assignment to a center slot. 

4. At Enrollment at a Center. Random assignment would occur after applicants enroll at 
a center. 

In choosing the point of random assignment, we balanced several, often competing, research and 
operational objectives: 

Maximizing the Ability to Address Useful Policy Questions. The point of random assignment 
determines the policy questions that can be addressed with a simple comparison of the means of 
outcome variables between the program and control groups. The later in the application process 
random assignment occurs, the greater our ability to address questions about the impact of Job Corps 
on those who enroll, without statistically modeling the decision to enroll. As estimating the impact 
of Job Corps on enrollees requires a minimum number of enrollees, not applicants, the later in the 



application process random assignment takes place, the smaller the sample required for a given level 
of precision. 

Minimizing Disruption to Normal Job Corps Application Procedures. Minimizing the 
disruption to normal Job Corps procedures is important for two reasons. First, the validity of the 
results depends on altering the program and its application procedures as little as possible. Second, 
as applicants from all OA agencies (with a few exceptions) will be subject to random assignment, 
it is important that random assignment is easy to implement. 

Minimizing Program Impacts Prior to Random Assignment. The process of applying for Job 
Corps may affect youth even before they enroll at a center. In anticipation of enrolling in Job Corps, 
youth may forgo other job or training opportunities or make other important decisions about their 
lives, and they are likely to have invested emotionally in the program. The earlier in the intake 
process random assignment takes place, the less likely that applicants would be affected by the 
application process prior to random assignment. 

Minimizing the impact of the program on applicants prior to random assignment is important 
for two reasons. First, the validity of our results would be compromised if control group youth were 
significantly affected by their contact with Job Corps. Second, the less they are affected by the 
program, the less applicants lose from being assigned to the control group. Reducing the negative 
impact on control group members is an important ethical consideration and makes the study more 
acceptable to program operators. 

Minimizing the Impact of Random Assignment on Applicants. The existence of random 
assignment reduces an applicant’s expectation of being able to participate in Job Corps. If random 
assignment occurs late in the application process, random assignment may itself affect the behavior 
of applicants. For example, an applicant whose expectation of entering Job Corps falls may be more 



likely to take other training or job opportunities than to wait to see if he or she is allowed to enroll 
in Job Corps. The ability to generalize our impact estimates would be compromised if the existence 
of the study altered applicants’ behavior. 

3. Rationale for the Chosen Point of Randomization 

We performed random assignment on all Job Corps applicants immediately after they were 
determined eligible for Job Corps and were ready for assignment to a center but before they were 
assigned to an open slot at a center. We believe this represented the best compromise among the 
different objectives. The resulting sample frame included only applicants who had been fully 
approved for the program after any routine court verifications, reviews by regional offices, and 
quality assurance checks on application folders. 

Performing random assignment after an application was approved had the following advantages: 

• It allowed us to address a useful and well-defined policy question: What are the impacts 
of Job Corps on eligible applicants? 

• It provided an opportunity to obtain information on the reasons that about 30 percent of 
eligible applicants fail to enroll at a center. 

• Program staff could incorporate random assignment into existing application procedures 
relatively easily— it became an additional check on eligibility. 

• Any impact the program had on applicants not yet determined to be eligible (and, thus, 
before random assignment) was likely to be small and short-term. 

• Random assignment at this early point in the application process was unlikely to have 
large effects on the behavior of applicants. 

• The point at which youth are found eligible for Job Corps could be defined consistently 
across different OA agencies and regions. 

We chose not to randomize when applicants first completed an application, because nearly 40 
percent of the sample would never enroll in Job Corps, as a result either of being found ineligible 



or of having lost interest. Thus, we would require a much larger sample to ensure precise estimates 
of the impact of Job Corps on those who enroll. 

We chose not to randomize after the applicant had been offered an open slot, because it could 
take months to assign an applicant to an open slot. By the time they have been offered an open slot, 
applicants, expecting to enroll in the program, may have invested a great deal in the program both 
emotionally and in forgone alternative opportunities. Random assignment itself may affect 
applicants’ behavior if it occurs this late in the process. A further disadvantage of this point is that 
because center assignment procedures vary slightly by region, the sample frame would not be 
consistent across regions. 

We rejected randomizing at enrollment at a center because it would involve an unacceptable 
burden on the control group youth who would have to be sent home after they arrived at a center, and 
it would be extremely disruptive to normal center operations. In addition, there would be sufficient 
time between application and enrollment at a center that the program could have an impact on the 
lives of youth prior to random assignment. 

B. CHOICE OF SAMPLE FRAME 

Early in the study design process, we decided that the sample frame should include all eligible 
applicants for Job Corps during an intake period of about one year. While we adhered to this basic 
decision, the sample frame was refined to account for issues that arose in implementing the study. 
In summary, the sample frame for the study consisted of all youth who met the following criteria: 

• Applied for Job Corps and were determined eligible for the program 

• Applied to Job Corps between November 17, 1994, and December 16, 1995, and were 
sent for random assignment on or before February 29, 1996 
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• Applied to Job Corps at any OA agency with a primary office located in the contiguous 
48 states or the District of Columbia 

• Had not previously participated in Job Corps (were not “readmits”) 

• Were not applicants for one of seven special Job Corps programs 

This section describes our rationale for choosing this sample frame. 

1. Clustered Versus Nonclustered Sample Design 

The Job Corps study is based on a fully national sample. With a few exceptions (discussed 
next), the members of the program and control groups were sampled from all OA agencies (including 
centers that recruit) in all parts of the United States, rather than from only some OA agencies in 
certain areas. We chose a fully national sample for two reasons. First, impact estimates made from 
a national sample will be more precise than estimates made from a clustered sample of the same size. 
Second, a nonclustered design spread the burden of random assignment across all OA agencies and 
Job Corps centers, reducing the burden on any one agency. 

2. Exclusions from the Sample Frame 

The sample frame comprised all youth who applied and were found eligible for Job Corps at any 
OA agency, with the following exceptions: (1) youth recruited outside the contiguous 48 states and 
the District of Columbia, (2) applicants who had previously participated in Job Corps, and (3) 
applicants for some special Job Corps programs. 

a. Youth Recruited Outside the Contiguous 48 States and the District of Columbia 

Applicants from four OA agencies-those with primary offices located in Alaska, Hawaii, Puerto 
Rico, and the Virgin Islands— were excluded from the sample frame because of the high costs of 
interviewing sample members and implementing random assignment outside the U.S. mainland. 3 

3 In program year 1995, about three percent of Job Corps center slots were outside the U.S. 

(continued...) 



